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Linear MMSE Estimation of Random Variables 


9.1.6 Linear MMSE Estimation of Random Variables 

Suppose that we would like to estimate the value of an unobserved random variable X, given 
that we have observed Y = y. In general, our estimate x is a function of y 

* = giy)- 

For example, the MMSE estimate of X given Y = y is 

g(y) = E[X\Y = y]. 

We might face some difficulties if we want to use the MMSE in practice. First, the function 
g(y) = E[X | Y = y] might have a complicated form. Specifically, ifX and Y are random 
vectors, computing E[X\ Y = y] might not be easy. Moreover, to find E[X\ Y = y] we need to 
know f x | y(y), which might not be easy to find in some problems. To address these issues, we 

might want to use a simpler function g(y) to estimate X. In particular, we might want g(y ) to be 
a linear function of y. 

Suppose that we would like to have an estimator for X of the form 

X L = g(Y) = aY+b, 

where a and b are some real numbers to be detennined. More specifically, our goal is to 
choose a and b such that the MSE of the above estimator 

MSE = E[(X-X l ) 2 ] 

is minimized. We call the resulting estimator the linear MMSE estimator. The following 
theorem gives us the optimal values for a and b. 


Theorem 9.1 

Let X and The two random variables with finite means and variances. Also, let p be the 
correlation coefficient of X and Y. Consider the function 

h(a, b) = E\(X- aY- b) 2 \. 


Then, 

1. The function h(a, b) is minimized if 
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* Cov(X, 7) 

“ = “ = Var(y) . b = b = EX- aEY. 

2. We have h(a*,b*) = ( 1 -p 2 )Var(V). 

3. 2s[(X- a* Y- b*)Y] = 0 (orthogonality principle). 

Proof: We have 

h(a, b ) = is[(X- a7- b) 2 ] 

= £[X 2 + a 2 Y 2 + b 2 - 2aXY- 2 bX+ 2abY] 

= £X 2 + a 2 EY 2 + b 2 ~ 2aEXY~ 2bEX+ 2abEY. 


Thus, h(a, b ) is a quadratic function of a and b. We take the derivatives with respect to a and b 
and set them to zero, so we obtain 


EY 2 • a + EY- b = EXY (9.4) 
EY- a + b = EX (9.5) 


Solving for a and b, we obtain 


* Cov(X, 7) ^ „ 

“ b'=EX-aEY. 


It can be verified that the above values do in fact minimize h(a, b ). Note that Eq uation 9.5 
implies that E[X- a* Y - b * ] = 0. Therefore, 


h(a*,b*) = E[(X-a*Y-b*) 2 ] 
= Var (X-a*Y~b*) 
= Var (X-a*Y) 


Var(V) + a * 2 Var(7) - 2 a * Cov(X, 7) 


= Var(V) + 


Var(V) 


Co w(X,Y)\ Co v(X,7) 

Var (7) 2 ^ Var(7) 

Cov(X, 7) 2 
Var(7) 


Cov(X, 7) 


= (1 - /j)Var(V). 


Finally, note that 


C[(X-a*7-h*)7] = EXY - a* EY 2 - b* EY 
= 0 (by Equation 9.4). 
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Note thatX = X- a* Y- b* is the error in the linear MMSE estimation of X given Y. From 
the above theorem, we conclude that 

E[X] = 0, 

E[XY\ = 0. 

In sum, we can write the linear MMSE estimator of X given Y as 



Cov(X, Y) 
Var(y) 


(Y~EY) + EX. 


If p ~ p(X, Y) is the correlation coefficient of X and Y, then Cov(X, Y) = pa yO Y , so the above 
formula can be written as 


„ P°x 

X L = -(7- EY) + EX. 

Oy 


Linear MMSE Estimator 


The linear MMSE estimator of the random variable X, given that we have observed Y, is given 
by 


Cov(X, Y) 

X, = - (Y~ EY) + EX 

1 Var(y) v 7 

P°x 


■{Y- EY) + EX. 


The estimation error, defined as X = X-X L , satisfies the orthogonality principle: 


E[X\ = 0, 

Cov(X, Y) = E[XY] = 0. 

The MSE of the linear MMSE is given by 

E[(X-X l ) 2 ] = E[X 2 ] = (1 ~p 2 )Var(X). 

Note that to compute the linear MMSE estimates, we only need to know expected values, 
variances, and the covariance. Let us look at an example. 


Example 9.8 
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1 

Suppose X ~ Uniformly, 2), and given X = x, Y is exponential with parameter A = 

a. Find the linear MMSE estimate of X given Y. 

b. Find the MSE of this estimator. 

c. Check that E\XY\ = 0. 

• Solution 

o We have 


Cov(X, Y) 

X, = - (Y~EY) + EX. 

L Var (Y) y J 


Therefore, we need to find EX, EY, Var(7), and Co\ (X, Y). First, note that we have 
3 

EX = - , and 


EY=E[E[Y\X]\ 

(law of iterated expectations) 

= E[X\ 

1 

(since Y\X ~ Exponential —)) 

3 


~ 2' 


EY 2 = E[E[Y 2 \X]\ 

(law of iterated expectations) 

il 

tii 

N> 

1 

(since Y\X ~ Exponential (—)) 

= Jj2 x 2 dx 


14 


“ T' 


Therefore, 



Var(7) = EY 2 - (EY) 2 


14 9 

T ~ 4 
29 
12 ' 


We also have 


https://www.probabilitycourse.com/chapter9/9_1_6_linear_MMSE_estimat_of_random_vars.php 


4/6 



9/18/2018 


Linear MMSE Estimation of Random Variables 


Thus, 


EXY=E[E[XY \X]\ 
EXY=E[XE[Y \X]\ 

= E[X-X | 



7 

3' 


(law of iterated expectations) 
(given X, Xis a constant) 

(since Y\X ~ Exponential (—)) 

X 


Cov(X, Y) = £[XF] - (EX)(EY) 
7 3 3 

~~ 3 ~ 2 ’ 2 

1 

~~ 12 ' 


a. The linear MMSE estimate of X given Y is 


b. The MSE oiX L is 


Cov(X, Y) 

X ^^^ (Y ~ EY)+EX 

29 \ 2)2 

Y 42 
^ 29 + 29' 


MSE = (1 - p 2 )Var(X). 


Since X ~ Uniformly, 2), VarfY) = —. Also 


Cov 2 (X, Y) 
Var(A)Var(7) 

1 

29' 


Thus, 
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MSE = 



1 

12 


7 

87' 


c. We have 


X = X-X T 


= X- 


Y 

29 


42 

29' 


Therefore, 


£[XT] = E | 

= £[AT| 


Y 

-VI 

29 

29 / . 

EY 2 

42 


- EY 

29 

29 


7 14 42 3 

~ 3 ~ 3 • 29 ~ 29 ’ 2 
= 0 . 


•r— previous 
next —> 
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